Multiple-choice tests (MCTs) have been used for many years to determine the presence or absence of a skill/trait/ability, but there are limitations to these types of tests. One limitation results from the impact of randomness, or “guessing” by the test-taker. For example, a four-option MCT item has a 25% chance of being answered correctly, even when the test-taker is guessing. Past efforts to suppress guessing by subtracting the number of incorrect answers from the number of correct answers, however, suffer from the possibility that a test-taker might hesitate to answer an item that the test-taker was less than completely sure of, even if the answer being considered was correct.
Another limitation is the inferred knowledge from a MCT. The amount of knowledge displayed by selecting a correct answer from a list of presented options is significantly less than the amount of knowledge demonstrated by providing a correct answer to a fill-in-the-blank (FITB) item where the test-taker supplies the answer without having the benefit of any prompts. Tests using FITB items have a different set of problems, however. For example, FITB items are thought to be challenging to machine score, as slightly misspelled answers are at risk of being marked as incorrect. These tests are time consuming to hand score and hand scoring is open to human error. In addition, FITB items answered incorrectly do not reflect the possibility that the test-taker might have been able to select the correct answer when presented in a MCT format.
Another concern is driven by the potential for testing bias, which might expose the testing entity and/or the test assessment entity to potential legal action. In cases where a set “cut” score is used in assessments and decision-making, even a small amount of bias might lead to incorrect judgments and biased results, thereby raising questions as to the validity of using such tests to make certain kinds of decisions.
The amount and types of information obtained in the testing sessions often fall short of what is needed to identify and address these concern for bias and validity.